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Distributed  Memory 

That  most  intriguing  aspect  of  human  memory:  its  persistence  in  spite  of 
continual  loss  of  individual  neurons  over  the  lifetime  of  the  indi- 

(l  2) 

vidual  has  led  many  workers  *  to  the  concept  of  distributed  memory. 

For  distributed  memory  (more  like  a  hologram  than  a  photograph)  possesses 
in  a  very  natural  way  the  property  of  relative  invulnerability  to  the  loss 
of  storage  units:  individual  memory  sites  hold  superimposed  information 
concerning  many  events.  In  order  to  obtain  a  single  event,  information 
must  be  gathered  from  many  sites.  Loss  of  Individual  units  decreases 
signal  to  noise  ratios  but  does  not  lose  items  of  information. 

Further,  in  contrast  to  modern  computers  that  perform  large  numbers  of 
sequential  operations  very  rapidly  and  very  accurately,  the  central  nervous 
system  works  slowly  and  probably  not  with  enormous  accuracy  on  the  level  of 
individual  units,  with  cycle  times  that  cannot  be  shorter  than  a  few 
milliseconds.  However  we  can  make  complex  decisions  in  small  parts  of  a 
second.  This  suggests  very  strongly  that  there  is  much  parallel  processing 
in  the  brain  -  an  idea  that  is  almost  obvious  on  inspection  of  a  component 
such  as  the  retina. 

It  is  now  commonly  thought  that  the  synaptic  Junction  may  be  a  means  to 
store  information  (memory,  for  example)  as  well  as  to  transmit  it  from 
neuron  to  neuron.  Large  networks  of  neurons  connected  to  other  neurons 
via  modifiable  synaptic  junctions  provide  the  physiological  substrate  for 
the  distributed  parallel  systems  discussed  here. 


» 


For  a  distributed  memory  it  is  the  simultaneous  or  near  simultaneous 
activities  of  many  different  neurons  (the  result  of  external  or  internal 
stimuli)  that  is  of  interest.  Thus  a  large  spatially  distributed  pattern 
of  neuron  discharges,  each  of  which  might  not  be  very  far  from  spontaneous 
activity,  could  contain  important,  if  hard  to  detect,  information.  Let  us 
consider  the  behavior  of  an  idealized  neural  network  (that  might  be 
regarded  as  a  model  component  of  a  nervous  system)  to  illustrate  some  of 
the  important  features  of  distributed  mappings. 

Consider  N  neurons  1,  2...N,  each  of  which  has  some  spontaneous  firing  rate 
rjo  (This  need  not  be  the  same  for  all  of  the  neurons  nor  need  it  be 
constant  in  time.)  We  can  then  define  an  N- tuple  whose  components  are  the 
difference  between  the  actual  firing  rate  rj  of  the  jth  neuron  and  the 
spontaneous  firing  rate  rjg 

fj'=  rj  -  rjo.  (1) 

By  constructing  two  such  banks  of  neurons  connected  to  one  another  (or  even 
by  the  use  of  a  single  bank  which  feeds  signals  back  to  itself),  we  arrive 
at  a  simplified  model  as  illustrated  in  Fig.  1. 


Fig.  1.  An  ideal  distributed  mapping.  Each  of 
the  N  input  neurons  in  F  is  connected  to  each 
of  the  N  output  neurons  in  G  by  a  single  ideal 
Junction.  (Only  the  connections  to  i  are  drawn. ) 

The  actual  synaptic  connections  between  one  neuron  and  another  are 
generally  complex  and  redundant;  we  idealize  the  network  by  replacing 
this  multiplicity  of  synapses  between  axons  and  dendrites  by  a  single 
ideal  junction  which  summarizes  logically  the  effect  of  all  of  the 
synaptic  contacts  between  the  incoming  axon  branches  from  neuron  j  in 
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4. 

the  F  bank  and  the  dendrites  of  the  outgoing  neuron  i  in  the  G  bank 
(Fig.  2). 


Fig.  2.  An  ideal  synaptic  junction 


Although  the  firing  rate  of  a  neuron  depends  in  a  complex  and  nonlinear 
fashion  on  the  presynaptic  potentials,  there  is  usually  a  resonably 
well  defined  linear  region  in  which  some  very  interesting  network 
properties  are  already  evident.  We  therefore  focus  our  attention  on 
the  region  above  threshold  and  below  saturation  for  which  the  firing 
rate  of  neuron  i  in  G,  g^t  is  mapped  from  the  firing  rates  of  all  of 
the  neurons  fj,  in  F  by: 

11 

«i  *  E  Aijfj.  (2) 

j-] 


In  doing  this  we  are  regarding  as  important  average  firing  rates,  and 
time  averages  of  the  instantaneous  signals  in  a  neuron  (or  perhaps  a 


small  population  of  neurons) .  We  are  further  using  the  known  integrative 
properties  of  neurons. 


We  may  then  regard  [A^ ] ,  the  synaptic  strengths  of  the  N  ideal  junctions, 
as  a  matrix  or  a  mapping  which  takes  us  from  a  vector  in  the  F  space  to  one 
in  the  G  space.  This  maps  the  neural  activities  f  =  ( f-| ,  f2...fN)  in  the  F 
space  into  the  neural  activities  g  =  (gi...gff)  in  the  G  space  and  can  be 
written  in  the  compact  form 


g  =  Af. 


(3) 


It  has  been  shown  that  the  non-local  mapping  A  can  serve  in  a  highly 

precise  fashion  as  a  memory  that  is  content  addressable  and  in  which 

'logic'  is  a  result  of  association  and  an  outcome  of  the  nature  of  the 
(2) 

memory  itself. 


To  illustrate  with  a  simple  example,  that  illuminates  the  content 
addressable  properties  of  the  mapping,  write: 


A  =  E  gw  x  f u  . 

V 


(4) 


i 

Here  gU  and  f u  are  output  and  input  patterns  of  neural  activity.  The 
symbol,  x,  represents  the  'outer'  product  between  the  input  and  output 
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vectors.  Although  (4)  is  a  well  known  mathematical  form,  its  meaning 
as  a  mapping  among  neurons  deserves  some  discussion.  The  ijth  element 
of  A  gives  the  strength  of  the  ideal  junction  between  the  incoming 
neuron  j  in  the  F  bank  and  the  outgoing  neuron  i  in  the  G  bank  (Fig. 

2). 

Since 

Aij  =  I  gi  fj.  (5) 

u 

the  ijth  junction  strength  is  composed  of  a  sum  of  the  entire 
connectivity  of  the  system  as  reflected  in  firing  rates  of  the  neurons 
connected  to  this  junction.  Each  association  however,  is  stored  over 
the  entire  array  of  N  x  N  junctions.  This  is  the  essential  meaning  of 
a  distributed  memory:  Each  event  is  stored  over  a  large  portion  of  the 
system,  while  at  any  particular  local  point  many  events  are 
superimposed. 

The  fundamental  problem  posed  by  a  distributed  memory  is  the  address 
and  accuracy  of  recall  of  the  stored  patterns. 

An  arbitrary  event,  e,  in  the  external  world  mapped  by  the  sensory 
apparatus  into  the  pattern  of  neural  activity,  f.  will  according  to  (3), 
generate  the  response,  g  =  Af,  in  G. 


(The  pattern,  f.  might  also  be  the  result  of  some  other  internal  pattern  of 
neural  activity.)  If  we  equate  recognition  with  the  strength  of  this 
response,  say  the  inner  product  (g,  g),  and  if,  for  illustration,  we  define 
separated  events  as  those  that  map  into  orthogonal  vectors:  el  f  1 ,  e2  f2 
...  ek  fK  where 

(f“  ,  f6  )  =  0  a  4  8  (6) 

1  a  =  8 

then  the  mapping  A  will  distinguish  between  those  events  it  contains, 

u 

the  f  ,  u  :  1,  2,  ...  K  and  other  events  separated  from  these. 

K  K 

Af°  “  2  gU  x  tv  fa  =  Z  gv  (fp  ,  fa) 

U-l  y=l 

s  ga  if  fa  is  one  of  the  vectors  f1...fK  (7) 

a 

s  0  if  f  is  not  one  of  these  vectors 


In  this  special  situation,  the  content  addressable,  parallel 
distributed  memory  is  as  precise  as  a  localized  memory. 

fl  -*■  gl 

f2  -  g2  (8) 

fk  gK 

The  properties  of  such  a  memory  in  more  general  circumstances,  its 

ability  to  form  associations,  to  map  the  external  world,  to  create  an 

( 2 ) 

'animal  logic'  have  been  discussed  elsewhere. 

Long  and  short-term  memory 

The  N2  junctions,  Ajj,  contain  the  content  of  the  distributed  memory. 
It  could  be  that  a  particular  junction  strength,  A±jf  is  composed  of 
several  different  components  with  different  lifetimes  thought  of  as 
corresponding  to  different  physiological  or  anatomical  effects  (e.g. , 
changes  in  numbers  of  presynaptic  vesicles,  changes  in  numbers  of 
postsynaptic  receptors.  Changes  in  Ca++  levels  and/or  availability, 
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anatomical  changes  such  as  might  occur  in  growth  or  shrinkage  of 
spines).  We  then  have  the  possibility  that  the  actual  memory  content 
(even  in  the  absence  of  additional  learning)  will  vary  with  time.  For 
a  two-component  system  we  might  have 


Aij(D  =  A (t)  ♦  4!hOTt\t).  (9) 


where  A ^ ^  represents  the  memory  at  some  time  t.  while  Af}°n8^and 

A-j-j  have  long  and  3hort  lifetimes.  Thus  in  time  A  ffho,rt^ill  decay, 
(t)  .  (long)  1 J 


leaving  AH 


=  A 


ij 


Whether  what  is  in  the  short-term  memory 


component  is  transferred  to  the  long-term  component  might  be  determined 
by  some  global  signal-  depending  on  the  interest  of  the  information 
contained  in  the  short-term  component. 


From  this  point  of  view  the  site  of  long  and  short-term  memory  can  be 
essentially  identical.  At  any  given  time  there  is  a  single  memory. 

The  distinction  between  long  and  short-term  memory  is  contained  in  the 
lifetime  of  the  different  components  of  A^j. 
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